A Summary and Comparison of Two Approaches for Determinization of Lattices

In many applications of speech and language processing, we generate intermediate results in the form of a lattice on which we apply finite-state operations. For example, we might POS tag the words in an ASR output lattice as an intermediate stage for language modeling. Currently, we have to convert the lattice into n-best scoring sub-lattices (one sub-lattice per unique input sequence), analyze each sub-lattice separately to get its best-scoring output sequence, and combine the resulting output sequences back into a lattice. We introduce two methods to eliminate the need for this unnecessary conversion by computing only the 1-best scoring output sequence for every input sequence in the lattice. This problem arises in any finite-state tagging task such as POS tagging, word segmentation, named entity recognition, as well as in discriminative training when we need to extract the best time boundaries, acoustic, pronunciation, or language model scores in an ASR lattice. Shafran et al. [1] and Povey et al. [4] independently proposed two solutions for the above problem. Obviously, selecting the n-best scoring sequences in the lattice is not a solution, because the result may contain more than one analysis for some input sequences while discarding all analyses of some other input sequences. Regular transducer determinization does not solve the problem either: for one thing, a transducer may include several different output sequences for a given input sequence, and still be deterministic as a transducer. Instead, in recently published work, Shafran et al. and Povey et al. define novel semirings in a way that determinization preserves only the best-scoring output sequence. In sections 2 and 3 we briefly introduce their methods. In section 4, we explain our ongoing research, which is a follow-up on the previous independent projects. We are investigating the similarities and differences between the two approaches and comparing them under the same conditions.